1. Technical Field
This disclosure relates to systems to and methods of processing digital images and more particularly to systems to and methods of generating an output image of enhanced quality and related systems and methods for correcting motion blur.
2. Description of the Related Art
Despite the great advances that have been made in the field of digital photography and CMOS/CCD sensors, several sources of distortion continue to be responsible for image quality degradation, among them noise and motion blur. See X. Liu and A. El Gamal, “Synthesis Of High Dynamic Range Motion Blur Free Image From Multiple Captures”, IEEE Transactions On Circuits And Systems, vol. 50, no. 4, April 2003. When an image is captured by a CMOS sensor, noise can be expressed as the sum of three kinds of noise sources, each of them related to the integration time T:
1. shot noise, S(T);
2. read out noise (inclusive of quantization noise), R(T);
3. reset noise, Z(T).
The collected charge C(T) is given (in a very simplified model, which does not take into account another common distortion of CMOS sensors, known as Fixed Pattern Noise (FPN)) by:
                              C          ⁡                      (            T            )                          =                                            ∫              o              T                        ⁢                                          (                                                      ph                    ⁡                                          (                      t                      )                                                        +                                      dc                    ⁡                                          (                      t                      )                                                                      )                            ⁢                                                          ⁢                              ⅆ                t                                              +                      R            ⁡                          (              T              )                                +                      Z            ⁡                          (              T              )                                                          (        1        )            wherein ph(t) and dc(t) are respectively the photocurrent and the dark current (current leakage produced independently from the presence of light). By supposing that the photocurrent is constant over the integration (exposure) time T, the Signal To Noise Ratio (SNR) can be expressed as:
                              s          ⁢                                          ⁢          n          ⁢                                          ⁢          r                =                  20          ⁢                                          ⁢                      log            10                    ⁢                                    ph              ·              T                                                                        (                                      ph                    +                    dc                                    )                                ·                T                            +                              σ                R                2                            +                              σ                Z                2                                                                        (        2        )            
From equation (2), an increase in the photocurrent or of the time T generally results in a SNR increase. Thus, longer exposure times usually lead to better image quality. On the other hand, a change in the photocurrent over time, due to motion, may lead to motion blur effects. In fact, in the presence of motion, the image formation process can be expressed as:C=RM+N  (3)wherein C is the image output at the camera side, R is the real scene, M is a transform matrix incorporating motion blur effects,  is the convolution operator and N is total sensor noise. This behavior is illustrated in FIG. 1, that depicts a short exposed image, free of motion blur, but relatively strongly corrupted by noise, and in FIG. 2, that represents a long exposed image, without noise, but with a noticeable blur.
Several techniques have been proposed in literature to reduce motion blur: hybrid imaging (M. B. Ezra and S. K. Nayar, “Motion Deblurring Using Hybrid Imaging”, IEEE Conference on Computer Vision and Pattern Recognition, 2003), through a camera able to register its own motion during capture which is used to evaluate and invert the matrix M; minimization techniques (A. R. Acha and S. Peleg, “Restoration of Multiple Images with Motion Blur in Multiple Directions”, IEEE Proceedings of the 5th Workshop On Applications of Computer Vision, 2000) to evaluate M from multiple blurred images; or using multiple captures, at different times and with different integration settings, to simultaneously extend dynamic range and reduce motion blur (X. Liu and A. El Gamal, “Synthesis Of High Dynamic Range Motion Blur Free Image From Multiple Captures”, IEEE Transactions On Circuits And Systems, vol. 50, no. 4, April 2003; X. Q. Liu and A. El Gamal, “Simultaneous Image Formation and Motion Blur Restoration via Multiple Capture”, International Conference on Acoustics, Speech, and Signal Processing, 2001). These techniques are primarily inspired by the use of multiple images, acquired at different instants, with the same integration (exposure) time for carrying out motion blur correction algorithms, while using de-noising techniques to prevent excessive image quality loss, as described in Z. Wei, Y. Cao and A. R. Newton, “Digital Image Restoration By Exposure Splitting And Registration”, Proceedings Of The 17th International Conference On Pattern Recognition, 2004. A general scheme illustrating a technique of this type is shown in FIG. 3.
According to this scheme, N shots, illustrated as Frames 1 to N, of a same scene are taken at 10. At 20, general video stabilization, they are then stabilized with a video stabilization algorithm by treating them as if they were a succession of images of a video sequence. At 30, an output image is generated by averaging the stabilized images.
Unfortunately, the quality of the digital images obtained with these techniques is not entirely satisfactory.